Skip to content

[PLUGIN-1844] Add label and use storage client for bucket cleanup. #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
itsankit-google merged 1 commit into from
Jun 3, 2025
Merged

[PLUGIN-1844] Add label and use storage client for bucket cleanup. #1552

itsankit-google merged 1 commit into from
Jun 3, 2025

Conversation

@itsankit-google
Copy link
Contributor

@itsankit-google itsankit-google commented Jun 2, 2025 •
edited
Loading

context : It is observed multiple times that temp GCS buckets are left undeleted after pipeline execution.

Tested in local sandbox

2025-06-02 20:36:34,463 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.u.BigQueryUtil@737] - Deleted GCS bucket 'bq-source-bucket-c533479c-eae1-449d-9ebe-1b29645ff395'.
2025-06-02 20:36:34,463 - DEBUG [SparkRunner-phase-1:i.c.p.g.c.GCPUtils@270] - Setting read timeout to 120 seconds.
2025-06-02 20:36:34,634 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.s.BigQuerySourceUtils@186] - Deleted temporary table '_E2E_SOURCE_54983dcc_6cbe_4bf1_9617_045b7b885495_99a24d64_6dc9_4e11_8112_eff4487c0492'
2025-06-02 20:36:35,411 - DEBUG [SparkRunner-phase-1:i.c.p.g.b.u.BigQueryUtil@737] - Deleted GCS bucket 'bq-sink-bucket-0c7abd5d-38a4-4ffa-b6b1-cbdc32898fd0'

image

@itsankit-google itsankit-google force-pushed the PLUGIN-1844 branch 4 times, most recently from 8e446fa to 166006e Compare June 2, 2025 18:12
@itsankit-google itsankit-google added the build Trigger unit test build label Jun 2, 2025
@itsankit-google itsankit-google force-pushed the PLUGIN-1844 branch 2 times, most recently from 9acfa88 to 9214ad3 Compare June 2, 2025 18:20
@itsankit-google itsankit-google force-pushed the PLUGIN-1844 branch 11 times, most recently from 3d46eba to a0da216 Compare June 2, 2025 22:47
@itsankit-google itsankit-google changed the title [PLUGIN-1844] Add support for cleanup of temp gcs buckets after pipeline completion [PLUGIN-1844] Add label and use storage client for bucket cleanup. Jun 2, 2025
@itsankit-google itsankit-google merged commit c13bece into develop Jun 3, 2025
15 of 16 checks passed
@itsankit-google itsankit-google deleted the PLUGIN-1844 branch June 3, 2025 08:27
This was referenced Jun 3, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sahusanket sahusanket sahusanket approved these changes

Assignees

No one assigned

Labels

build Trigger unit test build

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@itsankit-google @sahusanket